Papers with adapter-based PEFT
Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have ignited a surge in demand for effective fine-tuning techniques . data labeling is notoriously time-consuming and expensive, hindering the development of sizable labeled datasets . |
| Approach: | They propose to use active learning to reduce labeling costs by minimizing label complexity . they find PEFT adapter modules have significant potential in low-resource settings . |
| Outcome: | The proposed model outperforms FFT in low-resource settings and shows that it yields more stable representations of early and middle layers than FFT. |
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023.emnlp-main)
Copied to clipboard
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Lee
| Challenge: | Large language models (LLMs) have shown unprecedented performance across various tasks. |
| Approach: | They propose an easy-to-use framework that integrates adapters into LLMs . they evaluate adapters on 14 datasets from two different reasoning tasks . |
| Outcome: | The proposed framework can be used to fine-tune open-access language models with task-specific data and instruction data. |